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Abstract — Removal of specular reflection also called 
specularity or highlights from an image is very important before 
the image is processed or analyzed by computer for image 
processing, computer graphics and computer vision because it is 
considered as part of the image by the computer. Removal of 
specular reflection is important because when there are specular 
reflections in the image it causes errors in the results of many 
algorithms of computer vision and image processing. Due to 
specular reflections in images shapes in images get distorted 
therefore all shape detecting algorithms and algorithms based 
on shape detection, boundary detection results in erroneous 
outputs. Removal of specular reflection includes detection of the 
reflection or reflections in the given image and then changing 
those pixels with some other color pixel based on the other 
portions of the image which are not faulty. There exist many 
different techniques for the detection and removal of specular 
reflection. In this paper, some of such techniques and approach 
to tackle the problem are discussed. We have also proposed a 
simple yet efficient algorithm for specular reflection removal. 

Keywords — specularity, inpainting, bilateral filtering, high 
speed imaging. 


I. Introduction 

Specular reflection in the image has been a challenge in the 
field of computer vision and image processing from a very 
long period. It is defined as a type. They generally seem like 
surface features, but they are just artefacts that are caused due 
to changes in illumination from different angles [1]. 
Removing specular reflection can be seen as a problem of 
taking out information that is present in an image and then 
applying various transformation techniques to convert the 
information into meaningful representations. Specular 
reflection removal is quite important before image analysis 
and processing as applying processing algorithms like 
segmentation, recognition etc. to an image containing 
specular reflection may lead to significant inaccuracies in the 
results. It also reduces the robustness of certain algorithms 


and as a result, makes the algorithms less effective and 
reducing their applicability. They can also hide various 
defects in images which will remain hidden during inspection 
[2]. Specular reflection, especially on medical images like 
cervigrams, can result in misdiagnosis of cervical cancer or 
other types of diseases. So it can be seen easily that 
reflections in images cause a lot of problems ranging from 
making algorithms ineffective to life-threatening. Therefore 
it is crucial to develop effective algorithms and hardware 
solutions. 

One of the methods is to use a polarizing filter. But 
it is not convenient enough as not everyone possesses a 
polarizer every time they try to capture an image or take a 
photograph. Other Methods have also been studied like 
changing the position of the light source [3] or changing the 
position of the camera instead of a light source [4] in order to 
separate the specular reflection components. But, both the 
above-mentioned methods require the change in the position 
of the light source or the camera in the image processing 
system, thereby producing a huge strain or limitation when a 
picture is being taken. Here the speckle removal algorithms 
take the lead. In this paper, we discuss the various algorithms 
developed for specular reflection removal. 

II. Methods 

The papers selected were taken from various renowned 
publishing websites like IEEE, Springer, and Elsevier. The 
selected papers were then evaluated and papers with 
significant results and good accuracy were further taken into 
consideration. The papers were selected from the past 20 
years. The algorithms discussed in the paper mainly vary by 
virtue of their applications and implementations. Some 
algorithms perform better in situations when reflections are 
caused due to moisture on the surface, some perform better 
when reflections are caused due to the material of the object, 



some when reflections occur during medical imaging, while 
others perform better in general conditions. The images used 
in the paper are taken from Kaggle Intel/mobile ODT dataset 

[17]. 

III. Algorithms and Techniques 

A. Light Field Imaging Based Accurate Image Specular 
Highlight Removal 


components caused by the light source. This method also 
solves the problem intensity discontinuation due to image 
synthesis with the different light source. Experimental results 
are then used to show the efficiency and validity of the 
algorithm. 

C. Removal of Specular Reflection in Large Scale Ocean 
Surface Images 


Haoquian Wang et. al. have developed a Light Field imaging 
based image specular highlight removal algorithm [1]. Their 
algorithm works well in complex scenarios that occur in real 
life. This algorithm take advantage of the light field of 
Imaging Technology (Lytro ILLUM). The image is first 
captured (with specularity present) by the light field camera. 
Then depth of the image is accurately analysed and estimated 
and the specular pixels are then classified into two categories 
namely, "Unsaturated" and "Saturated" using a 
straightforward and concise thresholding strategy. For depth 
estimation and to achieve refocusing, utilisation of a depth 
estimation algorithm by combining/integrating both the 
correspondence and defocus cues has been done. 4D epipolar 
image (EPI) is exploited after deriving it from the LF data, 
making shears in order to operate refocusing. Then the 
responses of the two cues are computed using a single 
contrast-based approach presented in the paper. Both the 
locally estimated cues are then combined with a measure of 
confidence and global depth estimation is computed using 
MRFs to obtain the final result. At last, multiple views are 
subjected to color variance analysis and the two categories 
are used to recover diffused color information by conducting 
local color refinement individually on the two categories. The 
method is then experimentally evaluated by comparing them 
with the existing methods based on the light field dataset 
along with the standford light field archive, thus verifying its 
effectiveness. 


Shengke Wang et. al. used an image impainting technique for 
removal of specular reflection in large scale ocean surface 
images [7]. After detection of specular reflection in an image 
for removing reflection area image impainting technique 
proposed by Alexandru Telea based on Fast marching method 
is used [8]. In this algorithm filling of the specular reflection 
is done from the boundary pixels that decreases the size of 
area and repetition of filling boundary decreases the size of 
the area of specular reflection to zero. A pixel at the boundary 
is changed depending upon surrounding pixels of the image 
which are already correct. Effect of a surrounding pixel to the 
pixel to be corrected is based on the value of the surrounding 
pixel and its weight. Weight for a pixel is based on the 
distance between that pixel and the pixel to be corrected. 

, . = IggB e (p)W(p,<7)[/07)+ VR<?)(p-<?)] 
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Where I is impainting, p is the point to be inpainted. Bg is the 
surrounding region where q points exist, q are the points in 
the surrounding region that don’t contain specularity. W(p,q) 
is the weight of the points in Be. 

D. Removal of Specularities Using Color and Polarization 
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B. An Image-Correction Method for Specular Reflection 
Removal Using a High-speed Stroboscope 

Toshiaki Tsuji [5] suggests a method for separating diffused 
reflection components from specular reflection components. 
The author’s group developed a method of separating 
specular reflection components present in a high-speed video 
from diffused reflection components. A luminance variation 
(because of the flicker of a strobe) based estimation technique 
has been used. But the issue with this method is that a new 
specular reflection component is produced by the strobe. An 
algorithm is suggested to remove this specular reflection 
produced by the strobe. The arithmetic algorithm consists of 
the two processes mentioned below: 

• Specifying the highlight area 

• Compensation in the highlight area 

The original color of the object cannot be defined accurately 
as the highlight area is produced by strong specular reflection 


Shree K. Nayar et. al. proposed an algorithm for removal of 
specular reflection that uses polarization and color for 
separation of the two components of reflection, diffused and 
specular [9]. Most of the methods are either based on color or 
polarization. This algorithm applies new constraints on 
speculatrities by simultaneously using both color and 
polarization. Color of specular component is determined 
locally using polarization thereby constraining the diffused 
color at a pixel to a linear subspace of a single dimension. 
Neighbouring pixels whose color do not changes much with 
the pixel are found using this subspace. Information of diffuse 
color from consistent neighbours in subspace determines the 
value of the diffuse color of the pixel. This method of 
specular reflection removal can be used for specularity 
removal from direct source illuminations as well as 
interreflection between points in the scene. Neighbours 
having same diffuse color for each point is required for the 
algorithm to be used. The magnitude of neighbouring pixels 
may have a different magnitude of diffuse color but the 
direction in color space must be the same. The advantage of 
this algorithm over other methods is that the diffuse 
component in the image which is under the specularity region 
is not considered to be constant in the whole region. Another 
great advantage is that the diffuse component under the 
highlight region can be a texture and algorithm performs well 



in removing specularity. Reflection depends upon Fresnel 
ratio which is dependent on the angle of incidence and the 
material of the surface on which light is incident, this 
algorithm also performs well in case of reflection of different 
Fresnel ration in a particular specularity region. So, if in a 
region of specularity there are reflections due to different 
materials it can also be removed using this algorithm. 

E. Specular Reflection Reduction with Multi-Flash Imaging 


original image, such that the propagation of maximum diffuse 
chromaticity values is done from diffuse pixels to the 
specular pixels. Computation of diffuse pixel at each pixel is 
then done as a nonlinear function of the maximum diffused 
chromaticity that was estimated. If edge-preserving filters are 
used, this method can be extended for multi-color surfaces. 
Joint Bilateral filtering can be used to smoothen the 
maximum chromaticity omax,. As for the smoothing guidance, 
maximum diffuse chromaticity Amax is used. The maximum 
diffuse chromaticity can be defined as: 


Ramesh Raskar et. al. [10] proposed a technique to solve the 
problem of specular reflections. A camera with multiple 
flashes is used in this method. The camera will not be 
dependent on a single image of a scene rather it captures 
multiple images of a particular scene with different flash for 
each image. One flash at a time is kept in ON state and all the 
other flashes are kept in OFF state. This technique uses the 
phenomenon that position of specular reflection changes with 
the position and angle of incident light. So, there will be n 
number of images captured with n flashes, all at a different 
position that leads to specularity at a different position on 
each image. When captured images are combined, some 
highlights may overlap or may not overlap. This results in the 
three cases mentioned below: 


Amax = max (Ar, Ag, Ab). 

Recent developments in the field of fast bilateral techniques 
make the method run nearly 200 times faster than the state- 
of-the-art on the standard CPU. This technique is capable of 
processing high-resolution images at video rate and therefore 
is suitable for applications that work in real time. Also, the 
diffused reflections that are estimated will be locally smooth, 
does not result in noticeable artefacts. 

G. Correction Of Specular Reflection By Recursively 
Applying Smoothing Spatial Filter 


• Some highlights remain distinct (no overlapping) 

• Some highlights partially overlap. 

• Some highlights overlap completely. 

The proposed method works well with the first two cases but 
fails in the third case. It is to be noted that the 
boundaries/intensity edges around the highlights do not 
generally overlap even though specularities do overlap in 
input images. The principle idea of the approach is to take 
advantage of the variation of the gradient in n images, taken 
under n separate lighting conditions at the location of the 
given pixel, (x, y). If the mentioned pixel (x, y) is in the 
specular region in the case of no overlap or partial overlap, 
the gradient because of the specularity edges will be high in 
a minority or in only one of the n given images. A variable 
named h has been defined that represents an input image 
taken using a light source k , where k ranges from 1 to n. 


Generally, the detected specular reflections are considered as 
noise and we use smoothing spatial filter to either remove or 
smoothen this noise in the image. By this filter, the reflections 
can be blurred that is the filter covers places on the original 
image, and divides by the sum of all kernel elements, then 
multiplied with input pixel intensity and kernel values [11] 
and the formula for this filter is 


C= — {M(l)7 (1) + M(2)/(2) + -M(V)/(V) 


N 

C = ^Z {M(/c)/(/c)} 

k=1 


Here the c is renewed average and N is multiple of length and 
width [12]. 


This formula separates out the maximum intensity 
components from the image. These maximum intensity 
components are specularities. 

This formula calculates the median of pixels at 
location (x, y) of every image. 


The result will be an average value as it is divided by the 
multiple of length and width (N) 

To have a very smooth process we have to consider to 
consider mask elements as 1 and the above expression will be 
changed as 


F. Real-Time Specular Highlight Removal Using Bilateral 
Filtering: 

Qingxiong Yang et. al. [6] Suggests a method for real-time 
specular reflection removal based on the observation that 
maximum diffuse chromaticity changes quite smoothly in 
local patches. This property is used to then estimate the 
maximum diffuse chromaticity values by the usage of low 
pass filter. The low pass filter is applied directly to the 
maximum fraction of the color components present in the 


C = i{/( 1)+ /(2)+ •••/(«) 



k =1 


Therefore the final result of this filtering will be an average 
value as it divides with N after adding all intensities of the 


pixels. Our main task is to remove or minimize the specular 
reflections but after this process the image will be blurred as 
it is not our main aim we have to update the intensity of pixels 
present in the specular reflection area which is found through 
perception neural net as it was shown before but not change 
the whole intensity of the pixel. 


H. Automatic Detection Of Specular Reflections In The 
Cervix Images 

Here the specularities are detected in cervix images using 
intensity, saturation and gradient information and 
identification of highlighted region is done in a two stage 
segmentation process. First, coarse regions which have the 
reflections are found and second probabilistic modelling and 
segmentation are done to perfectly select the coarse regions. 
For the resulting region, a simple filling scheme is proposed. 

Highlights are most likely to found in the SR regions which 
are coarse regions, these specularities normally have low 
saturation values (S) and very high intensity (I) values, so 
initial regions can be found by using thresholds on I and S 

[13]. 
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Now a precise selection of previously defined coarse region 
is required for this they used Gaussian mixture model in 
which each pixel is considered as the 2-d vector and are these 
pixels are grouped as homogeneous regions and each region 
is represented as Gaussian distribution [14]. 
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For the final step, a filling scheme is used, which removes the 
strong gradients while making sure the original texture is not 
affected. 


preserving the original image structure, By the above 
collected information the pixel intensities can be shifted for 
separating the specular and diffuse image components. 
Mostly all the processes that remove the specular reflections 
normally have a little bit of image data lost in the process but 
here as there are thin nerves we cannot use normal methods 
we have to rather use a method that can collect as much as 
the information of the image by extending chromaticity [15]. 
Here I* 1 ^ is the diffuse component of reflectance and A- A 

r Ag A b 

It is clear that chromaticity of the diffuse pixels is always 
higher than that of the specular pixels, By making use the 
image pixel distribution projection into D space the diffuse 
reflectance component can be found as shown in the above 
formula and stored in a spatio-temporal volume. 


J. A Video Stream Processor For Real-Time Detection And 
Correction Of Specular Reflection In Endoscopic Images 

In this paper, the detection and correction of the specular 
reflections are by an algorithm called inpainting algorithm. 
Correcting a frame represents removing all the specularities 
that determined before and replacing them for 
neighbourhoods information, the common way is to use 
algorithms like Navier-stokes algorithm [16]. As it has many 
loops through a frame it requires large amounts of memory 
and computational power, so we use single frame memory 
architecture. It operates line by line 

1. A line of the frame is stored. 

2. We collect three data information, they are pixel 
value before the specular region P b , pixel value after 
specular region P c and the region’s width 

3. Now we can kind linear show a as: 

Pc-P b 

a = - 

w 

4. The leftmost pixel has index 0 in the specular region 
and po is given value P b + a. the new value of all other 
pixels are 


I. Removing Specular Reflection Components For Robotic 
Assisted Laparoscopic Surgery 

In this paper, they propose that at first all the chromatic 
information should be collected for the spatio-temporal 
volume so that this information can be used to remove the 
specular reflections on the epicardial surface of the heart 
while performing a robotic laparoscopic surgery while 


P [+1 — Pi + U 

As it only operates in the horizontal direction it is 
necessary to correct the vertical dimension. This is done 
with a smoothing window by changing modified pixels 
with the average of its neighbors and it may have a little 
delay but it can be considered negligible while the time 
of real operation. 







IV. Comparison Table 


Authors 

ALGORITHM 

TECHNIQUE USED 

ADVANTAGES 

Wang et. al. 

Light Field Imaging 
Based Accurate Image 
Specular Highlight 

Removal 

• Based on Light Field 
imaging technology 

• Works well in complex 
real-life scenarios 

Tsuji et. al. 

An Image-Correction 
Method for Specular 
Reflection Removal 

Using a High-speed 
Stroboscope 

• Luminance variation based 
estimation technique 

• Solves discontinuous 

intensity problem due to 
image synthesis with a 
light source 

Wang et. al. 

Removal of specular 
reflection in large scale 
ocean surface images 

• Image Inpainting based on 
Fast Marching Method 

• Works well with 

underwater images of 
oceans 

Nayar et. al. 

Removal of 
Specularities Using 
Color and Polarization 

• Based on color and 
polarization of surrounding 
pixels 

• Removes direct reflection 
as well as inter-reflection 

• Works well in images with 
texture. 

Raskar et. al. 

Specular Reflection 
Reduction with Multi- 
Flash Imaging 

• Uses multi-flash camera 
(hardware), the position of 
specularity changes with a 
change in angle of incidence 
of light. 

• Removes distinct 

highlights. 

• Removes partially 

overlapped highlights 

Yang et. al. 

Real-time Specular 

Highlight Removal 

Using Bilateral 

Filtering 

• Based on the observation 
that maximum diffuse 
chromaticity changes quite 
smoothly in local patches 

• Capable of processing 
high-resolution images at 
video rate. 

• Suitable for applications 
that work in real time. 

Lehman et. 
al. 

Correction of specular 
reflections by 
recursively applying a 
smoothing spatial filter 

• Using smoothing spatial 
filter to remove noise 

• The new image will not be 
blurry as the whole 
intensity of the pixel is 
updated. 

Gonzalez et. 
al. 

Automatic detection of 
specular reflections in 
the cervix images 

• Probabilistic modelling and 
segmentation 

• As the filling scheme is 
used original texture is not 
affected 

Bertalmio et. 
al. 

Removing Specular 
Reflection Components 
for 

Robotic Assisted 
Laparoscopic Surgery 

• Using chromatic information 
and store it in spatio- 
temporal volume 

• The original image is 
distorted in the process as 
only image data is 
collected. 

Lin et. al. 

A Video Stream 
Processor for Real-time 
Detection and 
Correction of Specular 
Reflections in 
Endoscopic Images 

• we use single frame memory 
architecture 

• It takes very less memory 
and doesn’t need high 
computational power 


Table 1 
















V. Algorithm 

Let I m be an image from the set of images S. We first 
transform our RGB image to HSV (Hue, Saturation, and 
Value) format. HSV format was chosen as it is closer to 
human vision. Hue is the tint or color potion expressed as 
number from 0 to 360 degrees. Saturation describes the 
amount of grey in particular color. Saturation varies the color 
from white to primary color in hue. Value is the 
brightness/intensity value of the resulting color. The 
saturation component s was extracted from I m (I ms ) which 
was observed to better separate/extract bright specular 
reflection portion than the other components. The objective 
is to highlight the specular component region of the image 
I ms and suppress the background or other unimportant 
regions. Thus the image I ms was then converted to black & 
white I bw , which makes the specular region black and other 
regions/background white. It was observed in certain 
cervigrams that there was some sort of black ring or boundary 
to be found around the specularity (which is very thin). The 
boundary needs to be removed from the images as it can 
affect the performance of the algorithm. To tackle this, the 
size of the black colored region in I bw was slightly increased 
so that the increase in the region will cover up the black 
boundary which is found around the specular reflections. 

The removal procedure: To remove the specularities 
present in the original image. For this a mask m of size n x n 
(the size of the mask can be changed according to the 
requirements) was created. The pixels P t j in the black region 
of the I bw were found out. A m nxn mask was created with the 
pixel P t j at its centre. 


Then repeat below mentioned process for each color 
channel Q of the image l m in RGB color scheme: 

• Traverse I bw pixel by pixel and wherever the non 
white pixels P t j are found (white in I m ‘s context), 
calculate the mean of all the corresponding non 
white pixels inside the mask in the original image 
I m ( Ay ) and channel C t . 

• Replace the pixel at (i,j) position in channel 
^i=(red,green,yellow) of image I m with A^j. 

This process is repeated for every pixel in the image I bw until 
every specularity is averaged out. The resultant image is I mr . 
After running the algorithm, it can easily verify the 
effectiveness and the degree to which the algorithm was 
effective. In order to compare the original image with the I mr 
we got after running the algorithm, we used earth mover’s 
distance (EMD) to compare the two images. Higher emd 
score signifies better results achieved, i.e. more specular 
reflection removed. 

The table 2. Shows the EMD score along with the images 
for which the score was calculated. 

Ay = (?n + P12 "1 -b Pm + P 21 + P22 3 -b p 2n 3 — 

+ Pnl + Pn 2 3-b P nn)/K 


Where Pij represents the non white pixels in the mask m nxn 
and K represents number of non white pixels. 
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Figure 1: Visual representation of algorithm 



After Algorithm Implementation 
































































VI. Results 



Table 2 




















Original Image 
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Figure 2: Process of reflection removal 


Conclusion 

In this paper, we have discussed various specular reflection 
removal algorithms. Some of these algorithms perform better 
in certain conditions while others are effective in general 
conditions. Some are more accurate while others are less 
accurate but require less computation power and perform 
their operation in very less time. Characteristics of various 
materials in the real world makes specular reflection 
inevitable. Currently, existing specular reflection removal 
methods are quite effective and achieve good separation of 
specular components, but are limited to their applicability 
conditions. Most of the techniques mentioned above rely on 
specific reflection model. This along with other problems like 
noise sensitivity make these algorithms less effective and 
reduce the range of their applicability. More accurate and 
robust algorithms are required which can overcome the 
limitations of the current methods. Our future work will 
involve developing a more general and robust algorithm for 
specular reflection removal which overcomes the limitations 
of most of these algorithms along with maintaining its 
accuracy. 
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